{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "source": [
    "## General information\n",
    "\n",
    "正确预测地震对于防止死亡和基础设施受损非常重要。在本次比赛中，我们尝试根据地震信号数据预测下一次实验室地震的剩余时间。训练数据代表一个巨大的信号，但在测试数据中，我们有许多单独的块，我们需要预测每个块的时间。\n",
    "\n",
    "这是本次比赛的第二个内核，这是第一个内容的链接。https://www.kaggle.com/kernels/notebooks/new?forkParentScriptVersionId=12760528&userName=buluslee\n",
    "\n",
    "在这个内核中，我将尝试创建更多有用的功能并生成更多数据用于训练\n",
    "\n",
    "![](https://i.cbc.ca/1.4972912.1547133821!/fileImage/httpImage/image.jpg_gen/derivatives/16x9_780/new-brunswick-earthquake.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_kg_hide-input": true,
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from tqdm import tqdm_notebook\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import NuSVR, SVR\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "pd.options.display.precision = 15\n",
    "\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "import time\n",
    "import datetime\n",
    "from catboost import CatBoostRegressor\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import StratifiedKFold, KFold, RepeatedKFold\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.linear_model import LinearRegression\n",
    "import gc\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from scipy.signal import hilbert\n",
    "from scipy.signal import hann\n",
    "from scipy.signal import convolve\n",
    "from scipy import stats\n",
    "from sklearn.kernel_ridge import KernelRidge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bc314853a64c3ee3b7b14dd2ad99c06e77e16561"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "train = pd.read_csv('/kaggle/input/LANL-Earthquake-Prediction/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据可视化--训练集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_uuid": "029a1f0d374a39d7fac9a7c81d417973bce1c884"
   },
   "outputs": [],
   "source": [
    "train_acoustic_data_small = train['acoustic_data'].values[::50]\n",
    "train_time_to_failure_small = train['time_to_failure'].values[::50]\n",
    "\n",
    "fig, ax1 = plt.subplots(figsize=(16, 8))\n",
    "plt.title(\"Trends of acoustic_data and time_to_failure. 2% of data (sampled)\")\n",
    "plt.plot(train_acoustic_data_small, color='b')\n",
    "ax1.set_ylabel('acoustic_data', color='b')\n",
    "plt.legend(['acoustic_data'])\n",
    "ax2 = ax1.twinx()\n",
    "plt.plot(train_time_to_failure_small, color='g')\n",
    "ax2.set_ylabel('time_to_failure', color='g')\n",
    "plt.legend(['time_to_failure'], loc=(0.875, 0.9))\n",
    "plt.grid(False)\n",
    "\n",
    "del train_acoustic_data_small\n",
    "del train_time_to_failure_small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "70a699101fd731a3d0963a7c71b8229510bfe67f"
   },
   "source": [
    "* 我们可以看到，通常声学数据在失败之前显示出巨大的波动，数据的性质是周期性的;\n",
    "* 另一个重点：视觉失败可以预测为信号的大幅波动跟随小信号值的情况。这可用于预测从0到高值的“time_to_failure”变化;\n",
    "* 我认为将段中信号的最大值与某个阈值（1000或2000）进行比较可能是有用的，但它不起作用;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8fbeddf32663d5a5dd3399c3ed19271bbc937c92"
   },
   "source": [
    "## 特征生成\n",
    "我创建了几组功能：\n",
    "\n",
    "* 通常的聚合：mean，std，min和max;\n",
    "* 绝对值和百分比值之间的平均值之间的平均差异;\n",
    "* 绝对最小和最大值;\n",
    "* 前面提到的第一个和最后一个10000和50000值的聚合 - 我认为这些数据应该是有用的;\n",
    "* 最大值到最小值及其差值也计算大于500的值（任意阈值）;\n",
    "* 此内核的分位数功能：https：//www.kaggle.com/andrekos/basic-feature-benchmark-with-quantiles\n",
    "* 此内核的趋势功能：https：//www.kaggle.com/jsaguiar/baseline-with-abs-and-trend-features\n",
    "* 来自此内核的滚动功能：https：//www.kaggle.com/wimwim/rolling-quantiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_uuid": "e75947cef3bff46114dd09ce164625871d54b30e",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#创建一系列简单的衍生特征\n",
    "rows = 150_000\n",
    "segments = int(np.floor(train.shape[0] / rows))\n",
    "\n",
    "def add_trend_feature(arr, abs_values=False):\n",
    "    idx = np.array(range(len(arr)))\n",
    "    if abs_values:\n",
    "        arr = np.abs(arr)\n",
    "    lr = LinearRegression()\n",
    "    lr.fit(idx.reshape(-1, 1), arr)\n",
    "    return lr.coef_[0]\n",
    "\n",
    "def classic_sta_lta(x, length_sta, length_lta):\n",
    "    \n",
    "    sta = np.cumsum(x ** 2)\n",
    "\n",
    "    # 转化数据类型\n",
    "    sta = np.require(sta, dtype=np.float)\n",
    "\n",
    "    # 复制\n",
    "    lta = sta.copy()\n",
    "\n",
    "    # 计算lta和sta\n",
    "    sta[length_sta:] = sta[length_sta:] - sta[:-length_sta]\n",
    "    sta /= length_sta\n",
    "    lta[length_lta:] = lta[length_lta:] - lta[:-length_lta]\n",
    "    lta /= length_lta\n",
    "\n",
    "    # 边界取0\n",
    "    sta[:length_lta - 1] = 0\n",
    "\n",
    "    # 通过将零值设置为微小浮点数，避免除以零\n",
    "    dtiny = np.finfo(0.0).tiny\n",
    "    idx = lta < dtiny\n",
    "    lta[idx] = dtiny\n",
    "\n",
    "    return sta / lta\n",
    "\n",
    "X_tr = pd.DataFrame(index=range(segments), dtype=np.float64)\n",
    "\n",
    "y_tr = pd.DataFrame(index=range(segments), dtype=np.float64, columns=['time_to_failure'])\n",
    "\n",
    "total_mean = train['acoustic_data'].mean()\n",
    "total_std = train['acoustic_data'].std()\n",
    "total_max = train['acoustic_data'].max()\n",
    "total_min = train['acoustic_data'].min()\n",
    "total_sum = train['acoustic_data'].sum()\n",
    "total_abs_sum = np.abs(train['acoustic_data']).sum()\n",
    "\n",
    "def calc_change_rate(x):\n",
    "    change = (np.diff(x) / x[:-1]).values\n",
    "    change = change[np.nonzero(change)[0]]\n",
    "    change = change[~np.isnan(change)]\n",
    "    change = change[change != -np.inf]\n",
    "    change = change[change != np.inf]\n",
    "    return np.mean(change)\n",
    "\n",
    "for segment in tqdm_notebook(range(segments)):\n",
    "    seg = train.iloc[segment*rows:segment*rows+rows]\n",
    "    x = pd.Series(seg['acoustic_data'].values)\n",
    "    y = seg['time_to_failure'].values[-1]\n",
    "    \n",
    "    y_tr.loc[segment, 'time_to_failure'] = y\n",
    "    X_tr.loc[segment, 'mean'] = x.mean()\n",
    "    X_tr.loc[segment, 'std'] = x.std()\n",
    "    X_tr.loc[segment, 'max'] = x.max()\n",
    "    X_tr.loc[segment, 'min'] = x.min()\n",
    "    \n",
    "    X_tr.loc[segment, 'mean_change_abs'] = np.mean(np.diff(x))\n",
    "    X_tr.loc[segment, 'mean_change_rate'] = calc_change_rate(x)\n",
    "    X_tr.loc[segment, 'abs_max'] = np.abs(x).max()\n",
    "    X_tr.loc[segment, 'abs_min'] = np.abs(x).min()\n",
    "    \n",
    "    X_tr.loc[segment, 'std_first_50000'] = x[:50000].std()\n",
    "    X_tr.loc[segment, 'std_last_50000'] = x[-50000:].std()\n",
    "    X_tr.loc[segment, 'std_first_10000'] = x[:10000].std()\n",
    "    X_tr.loc[segment, 'std_last_10000'] = x[-10000:].std()\n",
    "    \n",
    "    X_tr.loc[segment, 'avg_first_50000'] = x[:50000].mean()\n",
    "    X_tr.loc[segment, 'avg_last_50000'] = x[-50000:].mean()\n",
    "    X_tr.loc[segment, 'avg_first_10000'] = x[:10000].mean()\n",
    "    X_tr.loc[segment, 'avg_last_10000'] = x[-10000:].mean()\n",
    "    \n",
    "    X_tr.loc[segment, 'min_first_50000'] = x[:50000].min()\n",
    "    X_tr.loc[segment, 'min_last_50000'] = x[-50000:].min()\n",
    "    X_tr.loc[segment, 'min_first_10000'] = x[:10000].min()\n",
    "    X_tr.loc[segment, 'min_last_10000'] = x[-10000:].min()\n",
    "    \n",
    "    X_tr.loc[segment, 'max_first_50000'] = x[:50000].max()\n",
    "    X_tr.loc[segment, 'max_last_50000'] = x[-50000:].max()\n",
    "    X_tr.loc[segment, 'max_first_10000'] = x[:10000].max()\n",
    "    X_tr.loc[segment, 'max_last_10000'] = x[-10000:].max()\n",
    "    \n",
    "    X_tr.loc[segment, 'max_to_min'] = x.max() / np.abs(x.min())\n",
    "    X_tr.loc[segment, 'max_to_min_diff'] = x.max() - np.abs(x.min())\n",
    "    X_tr.loc[segment, 'count_big'] = len(x[np.abs(x) > 500])\n",
    "    X_tr.loc[segment, 'sum'] = x.sum()\n",
    "    \n",
    "    X_tr.loc[segment, 'mean_change_rate_first_50000'] = calc_change_rate(x[:50000])\n",
    "    X_tr.loc[segment, 'mean_change_rate_last_50000'] = calc_change_rate(x[-50000:])\n",
    "    X_tr.loc[segment, 'mean_change_rate_first_10000'] = calc_change_rate(x[:10000])\n",
    "    X_tr.loc[segment, 'mean_change_rate_last_10000'] = calc_change_rate(x[-10000:])\n",
    "    \n",
    "    X_tr.loc[segment, 'q95'] = np.quantile(x, 0.95)\n",
    "    X_tr.loc[segment, 'q99'] = np.quantile(x, 0.99)\n",
    "    X_tr.loc[segment, 'q05'] = np.quantile(x, 0.05)\n",
    "    X_tr.loc[segment, 'q01'] = np.quantile(x, 0.01)\n",
    "    \n",
    "    X_tr.loc[segment, 'abs_q95'] = np.quantile(np.abs(x), 0.95)\n",
    "    X_tr.loc[segment, 'abs_q99'] = np.quantile(np.abs(x), 0.99)\n",
    "    X_tr.loc[segment, 'abs_q05'] = np.quantile(np.abs(x), 0.05)\n",
    "    X_tr.loc[segment, 'abs_q01'] = np.quantile(np.abs(x), 0.01)\n",
    "    \n",
    "    X_tr.loc[segment, 'trend'] = add_trend_feature(x)\n",
    "    X_tr.loc[segment, 'abs_trend'] = add_trend_feature(x, abs_values=True)\n",
    "    X_tr.loc[segment, 'abs_mean'] = np.abs(x).mean()\n",
    "    X_tr.loc[segment, 'abs_std'] = np.abs(x).std()\n",
    "    \n",
    "    X_tr.loc[segment, 'mad'] = x.mad()\n",
    "    X_tr.loc[segment, 'kurt'] = x.kurtosis()\n",
    "    X_tr.loc[segment, 'skew'] = x.skew()\n",
    "    X_tr.loc[segment, 'med'] = x.median()\n",
    "    \n",
    "    X_tr.loc[segment, 'Hilbert_mean'] = np.abs(hilbert(x)).mean()\n",
    "    X_tr.loc[segment, 'Hann_window_mean'] = (convolve(x, hann(150), mode='same') / sum(hann(150))).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta1_mean'] = classic_sta_lta(x, 500, 10000).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta2_mean'] = classic_sta_lta(x, 5000, 100000).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta3_mean'] = classic_sta_lta(x, 3333, 6666).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta4_mean'] = classic_sta_lta(x, 10000, 25000).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta5_mean'] = classic_sta_lta(x, 50, 1000).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta6_mean'] = classic_sta_lta(x, 100, 5000).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta7_mean'] = classic_sta_lta(x, 333, 666).mean()\n",
    "    X_tr.loc[segment, 'classic_sta_lta8_mean'] = classic_sta_lta(x, 4000, 10000).mean()\n",
    "    X_tr.loc[segment, 'Moving_average_700_mean'] = x.rolling(window=700).mean().mean(skipna=True)\n",
    "    ewma = pd.Series.ewm\n",
    "    X_tr.loc[segment, 'exp_Moving_average_300_mean'] = (ewma(x, span=300).mean()).mean(skipna=True)\n",
    "    X_tr.loc[segment, 'exp_Moving_average_3000_mean'] = ewma(x, span=3000).mean().mean(skipna=True)\n",
    "    X_tr.loc[segment, 'exp_Moving_average_30000_mean'] = ewma(x, span=30000).mean().mean(skipna=True)\n",
    "    no_of_std = 3\n",
    "    X_tr.loc[segment, 'MA_700MA_std_mean'] = x.rolling(window=700).std().mean()\n",
    "    X_tr.loc[segment,'MA_700MA_BB_high_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr.loc[segment, 'MA_700MA_std_mean']).mean()\n",
    "    X_tr.loc[segment,'MA_700MA_BB_low_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr.loc[segment, 'MA_700MA_std_mean']).mean()\n",
    "    X_tr.loc[segment, 'MA_400MA_std_mean'] = x.rolling(window=400).std().mean()\n",
    "    X_tr.loc[segment,'MA_400MA_BB_high_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] + no_of_std * X_tr.loc[segment, 'MA_400MA_std_mean']).mean()\n",
    "    X_tr.loc[segment,'MA_400MA_BB_low_mean'] = (X_tr.loc[segment, 'Moving_average_700_mean'] - no_of_std * X_tr.loc[segment, 'MA_400MA_std_mean']).mean()\n",
    "    X_tr.loc[segment, 'MA_1000MA_std_mean'] = x.rolling(window=1000).std().mean()\n",
    "    X_tr.drop('Moving_average_700_mean', axis=1, inplace=True)\n",
    "    \n",
    "    X_tr.loc[segment, 'iqr'] = np.subtract(*np.percentile(x, [75, 25]))\n",
    "    X_tr.loc[segment, 'q999'] = np.quantile(x,0.999)\n",
    "    X_tr.loc[segment, 'q001'] = np.quantile(x,0.001)\n",
    "    X_tr.loc[segment, 'ave10'] = stats.trim_mean(x, 0.1)\n",
    "\n",
    "    for windows in [10, 100, 1000]:\n",
    "        x_roll_std = x.rolling(windows).std().dropna().values\n",
    "        x_roll_mean = x.rolling(windows).mean().dropna().values\n",
    "        \n",
    "        X_tr.loc[segment, 'ave_roll_std_' + str(windows)] = x_roll_std.mean()\n",
    "        X_tr.loc[segment, 'std_roll_std_' + str(windows)] = x_roll_std.std()\n",
    "        X_tr.loc[segment, 'max_roll_std_' + str(windows)] = x_roll_std.max()\n",
    "        X_tr.loc[segment, 'min_roll_std_' + str(windows)] = x_roll_std.min()\n",
    "        X_tr.loc[segment, 'q01_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.01)\n",
    "        X_tr.loc[segment, 'q05_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.05)\n",
    "        X_tr.loc[segment, 'q95_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.95)\n",
    "        X_tr.loc[segment, 'q99_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.99)\n",
    "        X_tr.loc[segment, 'av_change_abs_roll_std_' + str(windows)] = np.mean(np.diff(x_roll_std))\n",
    "        X_tr.loc[segment, 'av_change_rate_roll_std_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_std) / x_roll_std[:-1]))[0])\n",
    "        X_tr.loc[segment, 'abs_max_roll_std_' + str(windows)] = np.abs(x_roll_std).max()\n",
    "        \n",
    "        X_tr.loc[segment, 'ave_roll_mean_' + str(windows)] = x_roll_mean.mean()\n",
    "        X_tr.loc[segment, 'std_roll_mean_' + str(windows)] = x_roll_mean.std()\n",
    "        X_tr.loc[segment, 'max_roll_mean_' + str(windows)] = x_roll_mean.max()\n",
    "        X_tr.loc[segment, 'min_roll_mean_' + str(windows)] = x_roll_mean.min()\n",
    "        X_tr.loc[segment, 'q01_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.01)\n",
    "        X_tr.loc[segment, 'q05_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.05)\n",
    "        X_tr.loc[segment, 'q95_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.95)\n",
    "        X_tr.loc[segment, 'q99_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.99)\n",
    "        X_tr.loc[segment, 'av_change_abs_roll_mean_' + str(windows)] = np.mean(np.diff(x_roll_mean))\n",
    "        X_tr.loc[segment, 'av_change_rate_roll_mean_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_mean) / x_roll_mean[:-1]))[0])\n",
    "        X_tr.loc[segment, 'abs_max_roll_mean_' + str(windows)] = np.abs(x_roll_mean).max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6ad94de98384e95f8a769ce9670d8b55f636eec3"
   },
   "outputs": [],
   "source": [
    "print(f'{X_tr.shape[0]} samples in new train data and {X_tr.shape[1]} columns.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "25734169d50198f51f5fe6fff44caac7967d171d"
   },
   "source": [
    "### 采样\n",
    "\n",
    "* 我试图随机抽样150000行1k-10k次并将这些样本添加到训练数据中，但它严重降低了我的分数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b3e87358c38c3dc2a675be51a241d8bfa7171bbd"
   },
   "outputs": [],
   "source": [
    "np.abs(X_tr.corrwith(y_tr['time_to_failure'])).sort_values(ascending=False).head(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a2bd1c5cd7178d083b99b19a226e89486aa02d8b"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(44, 24))\n",
    "cols = list(np.abs(X_tr.corrwith(y_tr['time_to_failure'])).sort_values(ascending=False).head(24).index)\n",
    "for i, col in enumerate(cols):\n",
    "    plt.subplot(6, 4, i + 1)\n",
    "    plt.plot(X_tr[col], color='blue')\n",
    "    plt.title(col)\n",
    "    ax1.set_ylabel(col, color='b')\n",
    "\n",
    "    ax2 = ax1.twinx()\n",
    "    plt.plot(y_tr, color='g')\n",
    "    ax2.set_ylabel('time_to_failure', color='g')\n",
    "    plt.legend([col, 'time_to_failure'], loc=(0.875, 0.9))\n",
    "    plt.grid(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "02a2b2d6f64393ea928e60e4e660e47bec1afc16"
   },
   "outputs": [],
   "source": [
    "means_dict = {}\n",
    "for col in X_tr.columns:\n",
    "    if X_tr[col].isnull().any():\n",
    "        print(col)\n",
    "        mean_value = X_tr.loc[X_tr[col] != -np.inf, col].mean()\n",
    "        X_tr.loc[X_tr[col] == -np.inf, col] = mean_value\n",
    "        X_tr[col] = X_tr[col].fillna(mean_value)\n",
    "        means_dict[col] = mean_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a154abe0bd5d3e16b216d9375f09166aabba7d36"
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "scaler.fit(X_tr)\n",
    "X_train_scaled = pd.DataFrame(scaler.transform(X_tr), columns=X_tr.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bb0b51806d92a3e4c788e439ac17d5a5685fdfa8"
   },
   "source": [
    "### 读取测试集文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_uuid": "a1b46d0a370c3817a1cc98c022f9c62ebd135331",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "submission = pd.read_csv('/kaggle/input/LANL-Earthquake-Prediction/sample_submission.csv', index_col='seg_id')\n",
    "X_test = pd.DataFrame(columns=X_tr.columns, dtype=np.float64, index=submission.index)\n",
    "plt.figure(figsize=(22, 16))\n",
    "\n",
    "for i, seg_id in enumerate(tqdm_notebook(X_test.index)):\n",
    "    seg = pd.read_csv('/kaggle/input/LANL-Earthquake-Prediction/test/' + seg_id + '.csv')\n",
    "    \n",
    "    x = pd.Series(seg['acoustic_data'].values)\n",
    "    X_test.loc[seg_id, 'mean'] = x.mean()\n",
    "    X_test.loc[seg_id, 'std'] = x.std()\n",
    "    X_test.loc[seg_id, 'max'] = x.max()\n",
    "    X_test.loc[seg_id, 'min'] = x.min()\n",
    "        \n",
    "    X_test.loc[seg_id, 'mean_change_abs'] = np.mean(np.diff(x))\n",
    "    X_test.loc[seg_id, 'mean_change_rate'] = calc_change_rate(x)\n",
    "    X_test.loc[seg_id, 'abs_max'] = np.abs(x).max()\n",
    "    X_test.loc[seg_id, 'abs_min'] = np.abs(x).min()\n",
    "    \n",
    "    X_test.loc[seg_id, 'std_first_50000'] = x[:50000].std()\n",
    "    X_test.loc[seg_id, 'std_last_50000'] = x[-50000:].std()\n",
    "    X_test.loc[seg_id, 'std_first_10000'] = x[:10000].std()\n",
    "    X_test.loc[seg_id, 'std_last_10000'] = x[-10000:].std()\n",
    "    \n",
    "    X_test.loc[seg_id, 'avg_first_50000'] = x[:50000].mean()\n",
    "    X_test.loc[seg_id, 'avg_last_50000'] = x[-50000:].mean()\n",
    "    X_test.loc[seg_id, 'avg_first_10000'] = x[:10000].mean()\n",
    "    X_test.loc[seg_id, 'avg_last_10000'] = x[-10000:].mean()\n",
    "    \n",
    "    X_test.loc[seg_id, 'min_first_50000'] = x[:50000].min()\n",
    "    X_test.loc[seg_id, 'min_last_50000'] = x[-50000:].min()\n",
    "    X_test.loc[seg_id, 'min_first_10000'] = x[:10000].min()\n",
    "    X_test.loc[seg_id, 'min_last_10000'] = x[-10000:].min()\n",
    "    \n",
    "    X_test.loc[seg_id, 'max_first_50000'] = x[:50000].max()\n",
    "    X_test.loc[seg_id, 'max_last_50000'] = x[-50000:].max()\n",
    "    X_test.loc[seg_id, 'max_first_10000'] = x[:10000].max()\n",
    "    X_test.loc[seg_id, 'max_last_10000'] = x[-10000:].max()\n",
    "    \n",
    "    X_test.loc[seg_id, 'max_to_min'] = x.max() / np.abs(x.min())\n",
    "    X_test.loc[seg_id, 'max_to_min_diff'] = x.max() - np.abs(x.min())\n",
    "    X_test.loc[seg_id, 'count_big'] = len(x[np.abs(x) > 500])\n",
    "    X_test.loc[seg_id, 'sum'] = x.sum()\n",
    "    \n",
    "    X_test.loc[seg_id, 'mean_change_rate_first_50000'] = calc_change_rate(x[:50000])\n",
    "    X_test.loc[seg_id, 'mean_change_rate_last_50000'] = calc_change_rate(x[-50000:])\n",
    "    X_test.loc[seg_id, 'mean_change_rate_first_10000'] = calc_change_rate(x[:10000])\n",
    "    X_test.loc[seg_id, 'mean_change_rate_last_10000'] = calc_change_rate(x[-10000:])\n",
    "    \n",
    "    X_test.loc[seg_id, 'q95'] = np.quantile(x,0.95)\n",
    "    X_test.loc[seg_id, 'q99'] = np.quantile(x,0.99)\n",
    "    X_test.loc[seg_id, 'q05'] = np.quantile(x,0.05)\n",
    "    X_test.loc[seg_id, 'q01'] = np.quantile(x,0.01)\n",
    "    \n",
    "    X_test.loc[seg_id, 'abs_q95'] = np.quantile(np.abs(x), 0.95)\n",
    "    X_test.loc[seg_id, 'abs_q99'] = np.quantile(np.abs(x), 0.99)\n",
    "    X_test.loc[seg_id, 'abs_q05'] = np.quantile(np.abs(x), 0.05)\n",
    "    X_test.loc[seg_id, 'abs_q01'] = np.quantile(np.abs(x), 0.01)\n",
    "    \n",
    "    X_test.loc[seg_id, 'trend'] = add_trend_feature(x)\n",
    "    X_test.loc[seg_id, 'abs_trend'] = add_trend_feature(x, abs_values=True)\n",
    "    X_test.loc[seg_id, 'abs_mean'] = np.abs(x).mean()\n",
    "    X_test.loc[seg_id, 'abs_std'] = np.abs(x).std()\n",
    "    \n",
    "    X_test.loc[seg_id, 'mad'] = x.mad()\n",
    "    X_test.loc[seg_id, 'kurt'] = x.kurtosis()\n",
    "    X_test.loc[seg_id, 'skew'] = x.skew()\n",
    "    X_test.loc[seg_id, 'med'] = x.median()\n",
    "    \n",
    "    X_test.loc[seg_id, 'Hilbert_mean'] = np.abs(hilbert(x)).mean()\n",
    "    X_test.loc[seg_id, 'Hann_window_mean'] = (convolve(x, hann(150), mode='same') / sum(hann(150))).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta1_mean'] = classic_sta_lta(x, 500, 10000).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta2_mean'] = classic_sta_lta(x, 5000, 100000).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta3_mean'] = classic_sta_lta(x, 3333, 6666).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta4_mean'] = classic_sta_lta(x, 10000, 25000).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta5_mean'] = classic_sta_lta(x, 50, 1000).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta6_mean'] = classic_sta_lta(x, 100, 5000).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta7_mean'] = classic_sta_lta(x, 333, 666).mean()\n",
    "    X_test.loc[seg_id, 'classic_sta_lta8_mean'] = classic_sta_lta(x, 4000, 10000).mean()\n",
    "    X_test.loc[seg_id, 'Moving_average_700_mean'] = x.rolling(window=700).mean().mean(skipna=True)\n",
    "    ewma = pd.Series.ewm\n",
    "    X_test.loc[seg_id, 'exp_Moving_average_300_mean'] = (ewma(x, span=300).mean()).mean(skipna=True)\n",
    "    X_test.loc[seg_id, 'exp_Moving_average_3000_mean'] = ewma(x, span=3000).mean().mean(skipna=True)\n",
    "    X_test.loc[seg_id, 'exp_Moving_average_30000_mean'] = ewma(x, span=30000).mean().mean(skipna=True)\n",
    "    no_of_std = 3\n",
    "    X_test.loc[seg_id, 'MA_700MA_std_mean'] = x.rolling(window=700).std().mean()\n",
    "    X_test.loc[seg_id,'MA_700MA_BB_high_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X_test.loc[seg_id, 'MA_700MA_std_mean']).mean()\n",
    "    X_test.loc[seg_id,'MA_700MA_BB_low_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X_test.loc[seg_id, 'MA_700MA_std_mean']).mean()\n",
    "    X_test.loc[seg_id, 'MA_400MA_std_mean'] = x.rolling(window=400).std().mean()\n",
    "    X_test.loc[seg_id,'MA_400MA_BB_high_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] + no_of_std * X_test.loc[seg_id, 'MA_400MA_std_mean']).mean()\n",
    "    X_test.loc[seg_id,'MA_400MA_BB_low_mean'] = (X_test.loc[seg_id, 'Moving_average_700_mean'] - no_of_std * X_test.loc[seg_id, 'MA_400MA_std_mean']).mean()\n",
    "    X_test.loc[seg_id, 'MA_1000MA_std_mean'] = x.rolling(window=1000).std().mean()\n",
    "    X_test.drop('Moving_average_700_mean', axis=1, inplace=True)\n",
    "    \n",
    "    X_test.loc[seg_id, 'iqr'] = np.subtract(*np.percentile(x, [75, 25]))\n",
    "    X_test.loc[seg_id, 'q999'] = np.quantile(x,0.999)\n",
    "    X_test.loc[seg_id, 'q001'] = np.quantile(x,0.001)\n",
    "    X_test.loc[seg_id, 'ave10'] = stats.trim_mean(x, 0.1)\n",
    "    \n",
    "    for windows in [10, 100, 1000]:\n",
    "        x_roll_std = x.rolling(windows).std().dropna().values\n",
    "        x_roll_mean = x.rolling(windows).mean().dropna().values\n",
    "        \n",
    "        X_test.loc[seg_id, 'ave_roll_std_' + str(windows)] = x_roll_std.mean()\n",
    "        X_test.loc[seg_id, 'std_roll_std_' + str(windows)] = x_roll_std.std()\n",
    "        X_test.loc[seg_id, 'max_roll_std_' + str(windows)] = x_roll_std.max()\n",
    "        X_test.loc[seg_id, 'min_roll_std_' + str(windows)] = x_roll_std.min()\n",
    "        X_test.loc[seg_id, 'q01_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.01)\n",
    "        X_test.loc[seg_id, 'q05_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.05)\n",
    "        X_test.loc[seg_id, 'q95_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.95)\n",
    "        X_test.loc[seg_id, 'q99_roll_std_' + str(windows)] = np.quantile(x_roll_std, 0.99)\n",
    "        X_test.loc[seg_id, 'av_change_abs_roll_std_' + str(windows)] = np.mean(np.diff(x_roll_std))\n",
    "        X_test.loc[seg_id, 'av_change_rate_roll_std_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_std) / x_roll_std[:-1]))[0])\n",
    "        X_test.loc[seg_id, 'abs_max_roll_std_' + str(windows)] = np.abs(x_roll_std).max()\n",
    "        \n",
    "        X_test.loc[seg_id, 'ave_roll_mean_' + str(windows)] = x_roll_mean.mean()\n",
    "        X_test.loc[seg_id, 'std_roll_mean_' + str(windows)] = x_roll_mean.std()\n",
    "        X_test.loc[seg_id, 'max_roll_mean_' + str(windows)] = x_roll_mean.max()\n",
    "        X_test.loc[seg_id, 'min_roll_mean_' + str(windows)] = x_roll_mean.min()\n",
    "        X_test.loc[seg_id, 'q01_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.01)\n",
    "        X_test.loc[seg_id, 'q05_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.05)\n",
    "        X_test.loc[seg_id, 'q95_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.95)\n",
    "        X_test.loc[seg_id, 'q99_roll_mean_' + str(windows)] = np.quantile(x_roll_mean, 0.99)\n",
    "        X_test.loc[seg_id, 'av_change_abs_roll_mean_' + str(windows)] = np.mean(np.diff(x_roll_mean))\n",
    "        X_test.loc[seg_id, 'av_change_rate_roll_mean_' + str(windows)] = np.mean(np.nonzero((np.diff(x_roll_mean) / x_roll_mean[:-1]))[0])\n",
    "        X_test.loc[seg_id, 'abs_max_roll_mean_' + str(windows)] = np.abs(x_roll_mean).max()\n",
    "    \n",
    "    if i < 12:\n",
    "        plt.subplot(6, 4, i + 1)\n",
    "        plt.plot(seg['acoustic_data'])\n",
    "        plt.title(seg_id)\n",
    "\n",
    "for col in X_test.columns:\n",
    "    if X_test[col].isnull().any():\n",
    "        X_test.loc[X_test[col] == -np.inf, col] = means_dict[col]\n",
    "        X_test[col] = X_test[col].fillna(means_dict[col])\n",
    "        \n",
    "X_test_scaled = pd.DataFrame(scaler.transform(X_test), columns=X_test.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "759245da1d8ecdcda5f78cf6c866ef9ccb4f3aff"
   },
   "source": [
    "## 构建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b1e871cec60e445e82260cb6c3d7b553e290417a"
   },
   "outputs": [],
   "source": [
    "n_fold = 5\n",
    "folds = KFold(n_splits=n_fold, shuffle=True, random_state=11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_uuid": "b0f5152fc9a9a851838d0d1a9b134f3bcde03d0d"
   },
   "outputs": [],
   "source": [
    "def train_model(X=X_train_scaled, X_test=X_test_scaled, y=y_tr, params=None, folds=folds, model_type='lgb', plot_feature_importance=False, model=None):\n",
    "\n",
    "    oof = np.zeros(len(X))\n",
    "    prediction = np.zeros(len(X_test))\n",
    "    scores = []\n",
    "    feature_importance = pd.DataFrame()\n",
    "    for fold_n, (train_index, valid_index) in enumerate(folds.split(X)):\n",
    "        print('Fold', fold_n, 'started at', time.ctime())\n",
    "        X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]\n",
    "        y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n",
    "        \n",
    "        if model_type == 'lgb':\n",
    "            model = lgb.LGBMRegressor(**params, n_estimators = 50000, n_jobs = -1)\n",
    "            model.fit(X_train, y_train, \n",
    "                    eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric='mae',\n",
    "                    verbose=10000, early_stopping_rounds=200)\n",
    "            \n",
    "            y_pred_valid = model.predict(X_valid)\n",
    "            y_pred = model.predict(X_test, num_iteration=model.best_iteration_)\n",
    "            \n",
    "        if model_type == 'xgb':\n",
    "            train_data = xgb.DMatrix(data=X_train, label=y_train, feature_names=X.columns)\n",
    "            valid_data = xgb.DMatrix(data=X_valid, label=y_valid, feature_names=X.columns)\n",
    "\n",
    "            watchlist = [(train_data, 'train'), (valid_data, 'valid_data')]\n",
    "            model = xgb.train(dtrain=train_data, num_boost_round=20000, evals=watchlist, early_stopping_rounds=200, verbose_eval=500, params=params)\n",
    "            y_pred_valid = model.predict(xgb.DMatrix(X_valid, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n",
    "            y_pred = model.predict(xgb.DMatrix(X_test, feature_names=X.columns), ntree_limit=model.best_ntree_limit)\n",
    "        \n",
    "        if model_type == 'sklearn':\n",
    "            model = model\n",
    "            model.fit(X_train, y_train)\n",
    "            \n",
    "            y_pred_valid = model.predict(X_valid).reshape(-1,)\n",
    "            score = mean_absolute_error(y_valid, y_pred_valid)\n",
    "            print(f'Fold {fold_n}. MAE: {score:.4f}.')\n",
    "            print('')\n",
    "            \n",
    "            y_pred = model.predict(X_test).reshape(-1,)\n",
    "        \n",
    "        if model_type == 'cat':\n",
    "            model = CatBoostRegressor(iterations=20000,  eval_metric='MAE', **params)\n",
    "            model.fit(X_train, y_train, eval_set=(X_valid, y_valid), cat_features=[], use_best_model=True, verbose=False)\n",
    "\n",
    "            y_pred_valid = model.predict(X_valid)\n",
    "            y_pred = model.predict(X_test)\n",
    "        \n",
    "        oof[valid_index] = y_pred_valid.reshape(-1,)\n",
    "        scores.append(mean_absolute_error(y_valid, y_pred_valid))\n",
    "\n",
    "        prediction += y_pred    \n",
    "        \n",
    "        if model_type == 'lgb':\n",
    "            # feature importance\n",
    "            fold_importance = pd.DataFrame()\n",
    "            fold_importance[\"feature\"] = X.columns\n",
    "            fold_importance[\"importance\"] = model.feature_importances_\n",
    "            fold_importance[\"fold\"] = fold_n + 1\n",
    "            feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n",
    "\n",
    "    prediction /= n_fold\n",
    "    \n",
    "    print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n",
    "    \n",
    "    if model_type == 'lgb':\n",
    "        feature_importance[\"importance\"] /= n_fold\n",
    "        if plot_feature_importance:\n",
    "            cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n",
    "                by=\"importance\", ascending=False)[:50].index\n",
    "\n",
    "            best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n",
    "\n",
    "            plt.figure(figsize=(16, 12));\n",
    "            sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n",
    "            plt.title('LGB Features (avg over folds)');\n",
    "        \n",
    "            return oof, prediction, feature_importance\n",
    "        return oof, prediction\n",
    "    \n",
    "    else:\n",
    "        return oof, prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f368d0dc3a116451650e878e81da909ef4e942ae"
   },
   "outputs": [],
   "source": [
    "params = {'num_leaves': 128,\n",
    "          'min_data_in_leaf': 79,\n",
    "          'objective': 'huber',\n",
    "          'max_depth': -1,\n",
    "          'learning_rate': 0.01,\n",
    "          \"boosting\": \"gbdt\",\n",
    "          \"bagging_freq\": 5,\n",
    "          \"bagging_fraction\": 0.8126672064208567,\n",
    "          \"bagging_seed\": 11,\n",
    "          \"metric\": 'mae',\n",
    "          \"verbosity\": -1,\n",
    "          'reg_alpha': 0.1302650970728192,\n",
    "          'reg_lambda': 0.3603427518866501\n",
    "         }\n",
    "oof_lgb, prediction_lgb, feature_importance = train_model(params=params, model_type='lgb', plot_feature_importance=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1dbc733abb7ef3315410d987689bbc0c181d974a"
   },
   "outputs": [],
   "source": [
    "top_cols = list(feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n",
    "                by=\"importance\", ascending=False)[:50].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "389f82c089d500c80644862d7e55acdaf4bae508"
   },
   "outputs": [],
   "source": [
    "# Taking less columns seriously decreases score.\n",
    "# X_train_scaled = X_train_scaled[top_cols]\n",
    "# X_test_scaled = X_test_scaled[top_cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bd51243ddfa1b86b6f713f64c5005800380afa04"
   },
   "outputs": [],
   "source": [
    "oof_lgb, prediction_lgb, feature_importance = train_model(X=X_train_scaled, X_test=X_test_scaled, params=params, model_type='lgb', plot_feature_importance=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6f52e5cea8fea8f58c10e888616997fbbfe5c749"
   },
   "outputs": [],
   "source": [
    "xgb_params = {'eta': 0.03,\n",
    "              'max_depth': 9,\n",
    "              'subsample': 0.85,\n",
    "              'objective': 'reg:linear',\n",
    "              'eval_metric': 'mae',\n",
    "              'silent': True,\n",
    "              'nthread': 4}\n",
    "oof_xgb, prediction_xgb = train_model(X=X_train_scaled, X_test=X_test_scaled, params=xgb_params, model_type='xgb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "48a35d7fe14b3ab9c4778571928fce9b7c2cd042",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = NuSVR(gamma='scale', nu=0.9, C=10.0, tol=0.01)\n",
    "oof_svr, prediction_svr = train_model(X=X_train_scaled, X_test=X_test_scaled, params=None, model_type='sklearn', model=model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a3b4ec5e4a8a8b824a6f1a56106b1e8e7ab6ebc0"
   },
   "outputs": [],
   "source": [
    "model = NuSVR(gamma='scale', nu=0.7, tol=0.01, C=1.0)\n",
    "oof_svr1, prediction_svr1 = train_model(X=X_train_scaled, X_test=X_test_scaled, params=None, model_type='sklearn', model=model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f66e18dbce5690fde671799b809e40560991de90"
   },
   "outputs": [],
   "source": [
    "params = {'loss_function':'MAE'}\n",
    "oof_cat, prediction_cat = train_model(X=X_train_scaled, X_test=X_test_scaled, params=params, model_type='cat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5fecb618f872c6991afe86a8af17544ec8b76665"
   },
   "outputs": [],
   "source": [
    "model = KernelRidge(kernel='rbf', alpha=0.1, gamma=0.01)\n",
    "oof_r, prediction_r = train_model(X=X_train_scaled, X_test=X_test_scaled, params=None, model_type='sklearn', model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6adbcb019ff54fc86e276fc69392d24e8a6b46ec"
   },
   "source": [
    "### Stacking\n",
    "And now let's try stacking :) We can use the same function for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6a7caafd57c26da85fd62a73a0a98275c06e15fa"
   },
   "outputs": [],
   "source": [
    "train_stack = np.vstack([oof_lgb, oof_xgb, oof_svr, oof_svr1, oof_r, oof_cat]).transpose()\n",
    "train_stack = pd.DataFrame(train_stack, columns = ['lgb', 'xgb', 'svr', 'svr1', 'r', 'cat'])\n",
    "test_stack = np.vstack([prediction_lgb, prediction_xgb, prediction_svr, prediction_svr1, prediction_r, prediction_cat]).transpose()\n",
    "test_stack = pd.DataFrame(test_stack)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ec445fbb4ba84dbebbb73a62cb7d97294de37776",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "oof_lgb_stack, prediction_lgb_stack, feature_importance = train_model(X=train_stack, X_test=test_stack, params=params, model_type='lgb', plot_feature_importance=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7b997000c2a2aba8b51dad51b66d633afbf9adb9"
   },
   "source": [
    "Now let's see how do our models perform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_kg_hide-input": true,
    "_uuid": "29557b1a9c6619ddf71b11cd47ad5c6b720d51a0"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(18, 8))\n",
    "plt.subplot(2, 3, 1)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot(oof_lgb, color='b', label='lgb')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('lgb');\n",
    "plt.subplot(2, 3, 2)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot(oof_xgb, color='teal', label='xgb')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('xgb');\n",
    "plt.subplot(2, 3, 3)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot(oof_svr, color='red', label='svr')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('svr');\n",
    "plt.subplot(2, 3, 4)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot(oof_cat, color='b', label='cat')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('cat');\n",
    "plt.subplot(2, 3, 5)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot(oof_lgb_stack, color='gold', label='stack')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('blend');\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.suptitle('Predictions vs actual');\n",
    "plt.subplot(2, 3, 6)\n",
    "plt.plot(y_tr, color='g', label='y_train')\n",
    "plt.plot((oof_lgb + oof_xgb + oof_svr + oof_svr1 + oof_r + oof_cat) / 6, color='gold', label='blend')\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.title('blend');\n",
    "plt.legend(loc=(1, 0.5));\n",
    "plt.suptitle('Predictions vs actual');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2916c7f8c188b5455bfd55cbbfba1c72ca1ef86a"
   },
   "source": [
    "It turned out that stacking is much worse than blending on LB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c5d60a48d8c9071236815b0931111e86d20730fa"
   },
   "outputs": [],
   "source": [
    "submission['time_to_failure'] = (prediction_lgb + prediction_xgb + prediction_svr + prediction_svr1 + prediction_cat + prediction_r) / 6\n",
    "# submission['time_to_failure'] = prediction_lgb_stack\n",
    "print(submission.head())\n",
    "submission.to_csv('submission.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving features to csv, so that they can be easily used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_tr.to_csv('train_features.csv', index=False)\n",
    "X_test.to_csv('test_features.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
